Application No.: 10/727,609 
Attorney Docket No.: 24500-000008/US 

REMARKS 

Claims 1-24 are pending in the current application. Claims, 1, 13, 22, and 23 are 
independent claims. 

AMENDED DRAWINGS 
Applicant has amended Figures 6 and 7 to correct the misspelling of "register". No new 
matter has been added. 

SUBSTITUE SPECIFICATION 

Applicant has amended to the specification to correct the misspelling of "register" and to 
put the specification in better form for US practice. No new matter has been added by these 
changes to the specification. 

CLAIM OBJECTIONS 

The Examiner objects to claim 6 because of the following informality. Claim 6 recites "a 
shift resistor section" while Fig. 7-8 illustrate a shift register. The Examiner asserts that the 
Applicant intends to signify a shift register. 

Applicant has corrected claim 6 to recite "register" as recommended by the Examiner. 
Applicant submits that one skilled in the art understands that a shift register is composed of shift 
register sections. As such, one skilled in the art would also understand how to use sections of a 
shift register in the same manner as a single shift register. 

Applicant respectfiiUy requests the Examiner withdraw his objection. 



DESCRIPTION OF AN EXAMPLE EMBODIMENT 
In accordance with the present application, an example embodiment is directed to an 
apparatus and/or method for extracting objects fi-om an image, capable of performing object 
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recognition in real-time and/or having low memory requirements. Fig. 3 illustrates a method in 
accordance with one example embodiment. The example embodiment processes image data 
from a camera module 2 on a pixel-by-pixel basis (SI). The image data is converted from 
IJYfVY format to HSV format (S2). Thereafter, the system determines whether each pixel is a 
part of an object (S3) and removes any noise from the pixel data (S4). If the data is associated 
with an object (S5), then the object information is updated to store the information associated 
with the image data. Thereafter, the next pixel is processed (S8). 



CLAIM REJECTIONS-35 U.S.C, S 103 (Park) 

Claims 1, and 7-16 stand rejected under 35 U.S.C. § 103(a) as being unpatentable over 
Park (US 2002/0136449, hereinafter Park). Applicant respectfiiUy traverses this art grounds of 
rejection. 

Park is directed to an apparatus and method for extracting region information of objects 

queried in object extraction target images. Park discloses a method of extracting an object from 

an image, including the steps of: 

(a) receiving a query image including an object and an object 
extraction target image from which the object included in the query image 
is to be extracted; (b) determining a position of the object in the object 
extraction target image using pixel based color feature matching; (c) 
segmenting the query image and the object extraction target image into a 
pluraHty of regions using image features including color or texture; and (d) 
performing matching between the segmented regions in the query image 
and the segmented regions in the determined position of the object in the 
object extraction target image using color or texture features and 
determining a final object region using similarity in spatial adjacency 
between matching regions obtained as a result of the matching. 

(Park, paragraph [0011]) 

As discussed above, Park discloses a method of first identifying the location of a target 

object through color based feature matching ( step b). Park then discloses segmenting a query 
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image and a target image into a plurality of regions based on color or texture (step c). The 
regions are then compared based on color or texture and a final object region is determined based 
on spatial adjacency. 

As admitted by the Examiner, "Park does not explicitly teach retaining coordinate data of 
the pixel" (Office Action, Page 3). The Examiner asserts that although this is true, "it would have 
been obvious to one of ordinary skill... to utilize a feature of retaining coordinate data" because 
"faster retrieval time can be achieved" (Id). The Applicant disagrees. 

Applicant submits that the Examiner fails to establish a prima facie case of obviousness. 
The mere statement that Park could be modified does not render the resultant modification 
obvious to one of ordinary skill in the art unless Park also suggests the desirability of the 
modification. 

To establish obviousness based on a combination or modification of elements disclosed 
in the prior art, there must be some motivation, suggestion, or teaching of the desirabiUty of 
making the specific combination or modification that was made by the Applicants. The 
motivation suggestion or teaching may come explicitly from one of the following: 

(a) the statements in the prior art (patents themselves) 

(b) the knowledge of one of ordinary skill art, or in some cases, 

(c) the nature of the problem to be solved. ' 

In order to establish a prima facie case of obviousness under 35 U.S.C. §103(a), the 
Examiner must provide particular findings as to why the modification is obvious. The statement 
that "faster retrieval time can be achieved" is mere hand-waving. The Examiner should examine 



' See DembiczaK 50 USPQ at 1614 (Fed.Cir. 1999). 
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the claims based on the merits therein, and provide particular findings if any limitation is 
allegedly "obvious". 

Therefore, for all of the reasons discussed above, Park cannot disclose or suggest "an 
object information retaining section for retaining coordinate data of the pixel" as recited in 
independent claims 1 and 13. 

As such, claims 7-16, dependent upon independent claims 1 and 13, are likewise 
patentable over Park at least for the reasons given above with respect to independent claims 1 
and 13. 

Applicant respectfully requests that the Examiner withdraw this art grounds of rejection. 

CLAIM REJECTIONS - 35 V.S.C. S 103 (Park and Murao) 

Claims 3 and 23 stand rejected under 35 U.S.C. § 103(a) as being unpatentable over Park 
in view of Murao (US 6,728,406, hereinafter Murao). Applicant respectfully traverses this art 
grounds of rejection. 

As discussed above with regard to independent claim 1, Park cannot disclose or suggest 
"an object information retaining section for retaining coordinate data of the pixel" as recited in 
independent claim 1. 

A cursory review of Murao shows that Murao cannot make up for the deficiencies 
discussed above with respect to Park. Furthermore, assuming arguendo that Park and Murao 
could be combined (which applicant does not admit), the resuUing combination would still not 
disclose or suggest the limitations discussed above with regard to independent claim 1. 
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As such, claim 3, dependent upon independent claim 1, is likewise patentable over Park 
and Murao, alone or in any combination, at least for the reasons given above with respect to 
independent claim 1. 

Similarly, Park cannot disclose or suggest "storing coordinate data of the pixel" as recited 
in independent claim 23. 

A cursory review of Murao shows that Murao cannot make up for the deficiencies 
discussed above with respect to Park. Furthermore, assuming arguendo that Park and Murao 
could be combined (which applicant does not admit), the resulting combination would still not 
disclose or suggest the limitations discussed above with regard to independent claim 23. 

Therefore, independent claim 23 is patentable over Park (alone or in any combination 
with Murao) for all of the reasons set forth above. 

Applicant respectfully requests that the Examiner withdraw this art grounds of rejection. 

CLAIM REJECTIONS - 35 U.S.C. S 103 (Park and Kambavashi) 
Claim 2 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Park in view 
of Kambayashi (US 4,841,289 hereinafter Kambayashi). Applicant respectfully traverses this art 
grounds of rejection. 

As discussed above with regard to independent claim 1, Park cannot disclose or suggest 
"an object information retaining section for retaining coordinate data of the pixel" as recited in 
independent claim 1. 

A cursory review of Kambayashi shows that Kambayashi cannot make up for the 
deficiencies discussed above with respect to Park. Similarly, assuming arguendo that Park and 
Kambayashi could be combined (which applicant does not admit), the resulting combination 
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would still not disclose or suggest the limitations discussed above with regard to independent 
claim 1. 

As such, claim 2, dependent upon independent claim 1, is likewise patentable over Park 
and Kambayashi, alone or in any combination, at least for the reasons given above with respect 
to independent claim 1. 

Applicant respectfully requests that the Examiner withdraw this art grounds of rejection. 

CLAIM REJECTIONS - 35 U.S.C. S 103 (Park, Murao, and Lee) 
Claim 4 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Park in view 

of Murao in further view of Lee (US 6,828,982 hereinafter Lee). Applicant respectfully traverses 

this art grounds of rejection. 

As discussed above with regard to independent claim 1, Park cannot disclose or suggest 

"an object information retaining section for retaining coordinate data of the pixel" as recited in 

independent claim 1. 

As further discussed above, a cursory review of Murao shows that Murao cannot make up 
for the deficiencies discussed above with respect to Park. Similarly, Lee does not make up for the 
deficiencies discussed above with respect to Park and Murao. 

Assuming arguendo that Park, Murao, and Lee could be combined (which applicant does 
not admit), the resulting combination would still not disclose or suggest the limitations discussed 
above with regard to independent claim 1. 

As such, claim 4, dependent upon independent claim 1, is likewise patentable over Park, 
Murao, and Lee, alone or in any combination, at least for the reasons given above with respect to 
independent claim 1. 
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Applicant respectfully requests that the Examiner withdraw this art grounds of rejection. 

CLAIM REJECTIONS - 35 U.S.C, S 103 (Park and Sadaki) 

Claim 5 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Park in view 
of Sadaki (US 6,967,660 hereinafter Sadaki). Applicant respectfixUy traverses this art grounds of 
rejection. 

As discussed above with regard to independent claim 1, Park cannot disclose or suggest 
"an object information retaining section for retaining coordinate data of the pixel" as recited in 
independent claim 1. 

A cursory review of Sadaki shows that Sadaki cannot make up for the deficiencies 
discussed above with respect to Park. Similarly, assxmiing arguendo that Park and Sadaki could 
be combined (which applicant does not admit), the resulting combination would still not disclose 
or suggest the limitations discussed above with regard to independent claim 1. 

As such, claim 5, dependent upon independent claim 1, is likewise patentable over Park 
and Sadaki, alone or in any combination, at least for the reasons given above with respect to 
independent claim 1. 

Applicant respectfiilly requests that the Examiner withdraw this art grounds of rejection. 

CLAIM REJECTIONS - 35 U.S.C, S 103 (Park, Sadakh and Jeong) 

Claim 6 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Park in view 
of Sadaki in further view of Jeong (US 5,235,650 hereinafter Jeong). AppUcant respectfiilly 
traverses this art grounds of rejection. 



Page 18 



Application No.: 10/727,609 
Attorney Docket No.: 24500-000008/US 

As discussed above with regard to independent claim 1, Park cannot disclose or suggest 
"an object information retaining section for retaining coordinate data of the pixel" as recited in 
independent claim 1. 

As further discussed above, a cursory review of Sadaki shows that Sadaki cannot make 
up for the deficiencies discussed above with respect to Park. Similarly, Jeong does not make up 
for the deficiencies discussed above with respect to Park and Sadaki. 

Assuming arguendo that Park, Sadaki, and Jeong could be combined (which applicant 
does not admit), the resulting combination would still not disclose or suggest the limitations 
discussed above with regard to independent claim 1. 

As such, claim 6, dependent upon independent claim 1, is likewise patentable over Park, 
Sadaki, and Jeong, alone or in any combination, at least for the reasons given above with respect 
to independent claim 1 . 

Applicant respectfully requests that the Examiner withdraw this art grounds of rejection. 

CLAIM REJECTIONS - 35 U,S,C, S 103 (Park and Kamei) 

Claims 17-21, 22 and 24 stands rejected under 35 U.S.C. § 103(a) as being unpatentable 
over Park in view of Kamei (US 2001/0046309 hereinafter Kamei). Applicant respectfully 
traverses this art grounds of rejection. 

As discussed above, Park cannot disclose or suggest "an object information retaining 
section for retaining coordinate data of the pixel" as recited in independent claim 13. 

Similarly, Park cannot disclose or suggest "an object information retaining section for 
retaining coordinate data of the pixel" as recited in independent claim 22. 
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Furthermore, as discussed above, Park cannot disclose or suggest "storing coordinate data 
of the pixel" as recited in independent claim 23. 

A cursory review of Kamei shows that Kamei cannot make up for the deficiencies 
discussed above with respect to Park. Similarly, assuming arguendo that Park and Kamei could 
be combined (which applicant does not admit), the resuhing combination would still not disclose 
or suggest the limitations discussed above with regard to independent claims 13, 22, and 23. 

As such, claims 17-21 and 24 dependent upon independent claims 13 and 23, are likewise 
patentable over Park and Kamei, alone or in any combination, at least for the reasons given 
above with respect to independent claims 13 and 23. 

Applicant respectfully requests that the Examiner withdraw this art grounds of rejection. 

CLAIM REJECTIONS - 35 U.S.C. S 103 (Park. Murao, and Kamei) 

Claim 24 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Park in 
view of Murao in further view of Kamei (US 2001/0046309). Applicant respectfully traverses 
this art grounds of rejection. 

As discussed above with regard to independent claim 23, Park cannot disclose or suggest 
"storing coordinate data of the pixel" as recited in independent claim 23. 

As further discussed above, a cursory review of both Murao and Kamei shows that 
neither reference can make up for the deficiencies discussed above with respect to Park. 

Assuming arguendo that Park, Murao, and Kamei could be combined (which applicant 
does not admit), the resulting combination would still not disclose or suggest the limitations 
discussed above with regard to independent claim 23. 
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As such, claim 24, dependent upon independent claim 23, is likewise patentable over 
Park, Murao, and Kamei, alone or in any combination, at least for the reasons given above with 
respect to independent claim 23. 

Applicant respectfully requests that the Examiner withdraw this art groimds of rejection. 



Accordingly, in view of the above amendments and remarks, reconsideration of the 
objections and rejections and allowance of each claim in connection with the present application 
is earnestly solicited. 

Should there be any outstanding matters that need to be resolved in the present 
application, the Examiner is respectfully requested to contact Donald J. Daley at the telephone 
number of the undersigned below. 

If necessary, the Commissioner is hereby authorized in this, concurrent, and future 
replies, to charge payment or credit any overpayment to Deposit Account No. 08-0750 for any 
additional fees required under 37 C.F.R. § 1.16 or under 37 C.F.R. § 1.17; particularly, extension 
of time fees. 



CONCLUSION 



Respectfully submitted. 



HARNESS, DICKEY, & PIERCE, P.L.C. 




P.O. Box 8910 
Reston, Virginia 20195 
(703) 668-8000 



DJD/JRS:aeh 
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AMENDMENTS TO THE DRAWINGS 

The attached sheets of drawings include changes to Figure 6 and Figure 7. These sheets, 
which include Figures 6 and Figure 7, replace the original sheets including Figures 6 and Figure 7. 



Attachment: Replacement Sheets 
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ATION PROCESSING APPARATUS. IMAGE PROCESSING 
AME APPARATUS AND IMAGE PROCESSING METHOD 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

rOOOn The present invention relates to an object information processing 
apparatus for processing input image data to obtain information about an object 
contained in the image, an image processing system having a function of recognizing 
the object contained in the image using the object information, and a game 
apparatus and an image processing method using the image processing system. 

2. Description of the Related Art 

r00021 As high-performance and low-cost image pickup devices (e.g., 
CCDs, CMOS imagers, etc.), image processing DSP devices, and the like have been 
developed, a number of products have incorporated a camera module. Recently, 
camera modules are often employed even In mobile devices (e.g., cellular 
telephones, etc.), toys, peripheral devices thereof, and the like. 

r00031 Camera modules are utilized to input image data. Such an image 
data Input method can handle a large amount of information simultaneously and 
therefore is considerably efficient compared to data input methods which employ a 
key board, a jog dial, a joystick or the like to input operating commands. 



r00041 However, in conventional image data input methods using a 
camera module, input image data is typically displayed as an input image on a 
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monitor screen, and subsequently a final process, i.e., various judgnnent (store or 
send data, etc.) is performed by a human. 

roOOSI In some game apparatus or the like, the user performs actual 
movements in order to enhance virtual reality. To this end, a number of techniques 
for sensing movements have been developed. Particularly, if the movements of the 
user can be input as image data into an image processing system so that the 
movements can be recognized, this technique may be useful for game apparatuses. 

rOOOSI After the movements of the user have been input as image data 
to an image processing system, the central processing unit (CPU) of a computer 
needs to be used to eventually recognize the movements of the user. Before the 
eventual recognition, it is necessary to extract required information from the image 
data. Conventionally, a number of techniques for extracting an object from image 
data have been proposed. 

f00071 For example, an object recognition apparatus for recognizing a 
moving object has been proposed in Japanese Laid-Open Publication No. 6-266841 . 
In this technique, a stationary object is detected in image data by feature extraction; 
background data is obtained; and the background data is subtracted from current 
image data to recognize a moving object. 



[00081 Japanese Laid-Open Publication No. 63-88684 discloses an 
object recognition method, in which a luminosity gradient vector is calculated based 
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on the luminosity concentrations of adjacent pixels, and a difference in angle of the 
luminosity gradient vector is used to recognize an object. 

r00091 Japanese Laid-Open Publication No. 59-132075 discloses an 
image processing apparatus, in which only the contour of an object is stored in an 
image memory before recognition is performed by a central processing unit in order 
to reduce a processing time required for recognition. 

[00101 The above-described conventional object recognition apparatus 
(Japanese Laid-Open Publication No. 6-266841) requires a memory for storing a 
large amount of data for each step of object feature extraction, stationary object 
recognition, background data calculation, image subtraction, and moving object 
recognition. These steps require a large computation amount. Moreover, if real-time 
processing is required for the step, a high-performance CPU or processing operation 
circuit needs to be employed. 

room Also in the conventional object recognition method (Japanese 
Laid-Open Publication No. 63-88684), image data needs to be stored in order to 
compare image data between adjacent pixels during processing operations, and 
therefore, an image memory for storing at least one frame of image data is required. 

r001 21 Also in the conventional image processing apparatus (Japanese 
Laid-Open Publication No. 59-132075), data representing the contour of an object is 
stored in a memory before object recognition is performed by a CPU, and therefore, 
it is not easy to perform object recognition in real time. Particularly, real-time object 
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recognition cannot be achieved by a CPU having poor processing performance or a 
one-chip microcomputer incorporating the CPU are used. In the image processing 
apparatus, an image memory for storing data representing the contour of an object is 
required, so that it is not easy to reduce cost. 

r00131 In the above-described object recognition techniques, 
sophisticated processing operations are performed using an image memory to carry 
out complicated extraction for object recognition. Therefore, a memory for storing a 
large amount of image data for image processing and a CPU or processing operation 
circuit for processing the data at high speed are required, so that it is not easy to 
achieve an image processing system at low cost. Therefore, use of image data for 
data input is not suitable for inexpensive products, such ais toys or the like, although 
it is effective as a data input method. 

r00141 The variety and quantity of object extraction information obtained 
by conventional object recognition techniques are large, information required for 
object recognition varies depending on the image processing system. 

SUMMARY OF THE INVENTION 

rOOISI According to an aspect of the present invention, an object 
information processing apparatus for obtaining object information from input image 
data comprising pixels is provided, which comprises: an object determination section 
for determining whether or not each pixel is a part of an object to be extracted by 
comparing color information indicating a color of the pixel with a predetermined 
reference value for the object; and an object information retaining section for 
retaining coordinate data of the pixel as the object information if the pixel has been 
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determined by the object determination section to be a part of the object to be 
extracted. 

r00161 In one embodiment of this invention, the object determination 
section comprises: a first to an n^^ comparison sections for determining whether or 
not the color information of each pixel of the input image satisfies a first to an n^^ 
object conditions, respectively (n: natural integer); and an AND circuit for receiving n 
outputs of the first to the n*^ comparison sections. 

r00171 In one embodiment of this invention, the object information 
processing further comprises: an image data conversion section provided before the 
object determination section, for converting an UYVY value of the input image data 
to an HSV value. The object determination section compares the HSV value of each 
pixel output by the image data conversion section with the predetermined reference 
value to determine whether or not the pixel is a part of the object to be extracted. 

r00181 In one embodiment of this invention, the image data conversion 
section has a first conversion table for converting an UV value of the input image 
data to an H (hue) value, and a second conversion table for converting the UV value 
of the input image data to an 8 (saturation) value, and the image data conversion 
section outputs a Y value of the input image as a V (lightness) value. 

r00191 In one embodiment of this invention, the object information 
processing apparatus further comprises: a noise removal section for removing noise 
from a result of determination by the object determination section. . 
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r00201 In one embodiment of this invention, the noise removal section 
comprises: a shift resistor section for successively retaining the result of 
determination by the object determination section; and a noise removal 
determination section for determining whether or not a predetermined number or 
more of a plurality of results of the shift resistor section are equal to one another to 
determine whether or not the plurality of results are noise. 

r002n In one embodiment of this invention further comprises: an object 
inclusion relation determination section for determining whether a pixel of the input 
image data which has been determined by the object determination section to be a 
part of an object to be extracted is a part of an already detected object or of a new 
object which has not been detected, and generating coordinate data of the pixel. The 
object information retaining section retains the coordinate data generated by the 
object inclusion relation determination section for each detected object. 

[00221 In one embodiment of this invention, the object inclusion relation 
determination section generates four coordinate points: coordinates having 
maximum X, coordinates having minimum X, coordinates having maximum Y, and 
coordinates having minimum Y, where coordinates of the object are (X, Y). 

[00231 In one embodiment of this invention, when a pixel of the input 
image data appears which has been determined by the object determination section 
to be a part of an object to be extracted, the object inclusion relation determination 
section determines whether or not there is another pixel satisfying the same object 
condition, and when there is another pixel satisfying the same object condition, the 
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other pixel is cleternnined to be a part of the object and the coordinate data (X, Y) of 
the object is updated. 

f00241 In one embodiment of this invention, when a pixel of the input 
image data appears which has been determined by the object determination section 
to be a part of an object to be extracted and the object inclusion relation 
determination section determines that there is no pixel satisfying the same object 
condition, the pixel is determined to be a part of a newly detected object and 
information about the pixel is stored in the object information retaining section 
corresponding to the newly detected object. 

[00251 In one embodiment of this invention, when a plurality of object 
extraction conditions are provided, the object information retaining section retains a 
condition matching flag indicating which object condition is satisfied as a part of the 
object information. 

[00261 In one embodiment of this invention, the object information 
retaining section retains one frame of object information of an object which has been 
determined to be an object to be extracted. 

[00271 According to another aspect of the present invention, an image 
processing system is provided, which comprises: the above-described object 
information processing apparatus; an image data output apparatus for outputting 
image data into the object information processing apparatus; and a control apparatus 
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for controlling the object information processing apparatus and the image data output 
apparatus. 

r00281 In one embodiment of this invention, the image data output 
apparatus is provided with an image pickup device for taking an object image, and 
coordinate data of the object indicating a location of the object is coordinate data on 
the image pickup device. 

r00291 In one embodiment of this invention, the control apparatus 
comprises a processing operation section for reading out object information stored in 
the object information processing apparatus and performing a processing operation 
for recognizing an object contained in image data. 

r00301 In one embodiment of this invention, the processing operation 
section reads out the object information from the object information processing 
apparatus on a frame-by-frame basis. 

r003n In one embodiment of this invention, the processing operation 
section reads out the object information, which has been extracted on a frame-by- 
frame basis, from the object information processing apparatus and compares the 
object information between frames to detect movement or change of an object. 



r00321 In one embodiment of this invention, the control apparatus 
recognizes that the object is moving in a predetermined direction when coordinate 
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data of the object is changed in the predetermined direction between each of a 
plurality of consecutive frames. 

[00331 In one embodiment of this invention, the control apparatus 
recognizes that the object is looming toward a viewing site when a coordinate 
location of the object is not changed and a size of the object is expanding between 
each of a plurality of consecutive frames in coordinate data of the object. 

[00341 In one embodiment of this invention, the control apparatus 
recognizes that the object is moving away from a viewing site when a coordinate 
location of the object is not changed and a size of the object is shrinking between 
each of a plurality of consecutive frames in coordinate data of the object. 

[00351 In one embodiment of this invention, when the object has at least 
two colors, the control apparatus recognizes a behavior of the object when each 
color of the object is moved between each of a plurality of consecutive frames. 

[00361 According to another aspect of the present invention, a game 
apparatus, which recognizes a behavior of an object using the above-described 
image processing system, is provided. 

[00371 According to another aspect of the present invention, an image 
processing method is provided, which comprises the steps of: outputting image data 
from an image data output apparatus to an object information processing apparatus; 
converting a UYVY value of the input image data to an HSV value using the object 
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infornnation processing apparatus, comparing the HSV value of each pixel with a 
reference value provided for an object to be extracted to determine whether or not 
the pixel is a part of the object to be extracted, and storing coordinate data of the 
pixel which has been determined to be a part of the object to be extracted as object 
information on an object-by-object basis; and reading the object information stored in 
the object information processing apparatus using a processing operation section of 
a control apparatus on a frame-by-frame basis and recognizing an object contained 
in image data based on the object information. 

r00381 In one embodiment of this invention, the processing operation 
section of the control apparatus reads out the object information, which has been 
extracted on a frame-by-frame basis, from the object information processing 
apparatus and compares the object information between each frame to detect 
movement or change of an object. 

r00391 Functions of the present invention will be described below. 

[00401 According to the present invention, image data from the image 
data output section (e.g., a camera module, etc.) is serially processed by the object 
information processing apparatus on a pixel-by-pixel basis, and the coordinate data 
of an object to be extracted is stored in the object information storing section of the 
object information processing apparatus. As is different from conventional 
techniques, image data itself is not stored in a memory. Only the coordinate data of 
an object to be extracted is stored. Therefore, in the present invention, a large- 
capacity memory, such as a frame memory, a line memory or the like, is not 
required. Object information, such as coordinate data or the like, stored in the object 
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information storing section is rewritten as appropriate. In some cases, all data may 
be rewritten in the object information storing section. Therefore, it is preferable that a 
resistor in which all bits can be simultaneously accessed is used instead of a 
memory device, such as RAM or the like. 

r004n According to the present invention, an object contained in an 
image is extracted by comparing a value indicated by color information contained in 
image data with a reference value using the object information processing apparatus. 
Therefore, as is different from conventional techniques, complicated processing flow 
using a CPU for object extraction or the like is not required. The coordinate data of 
an extracted object can be used to recognize the object contained in an image using 
the processing operation section, such as a CPU, a processing operation circuit or 
the like. Therefore, image processing load on the image processing operation 
section can be significantly reduced. 

[00421 Therefore, according to the present invention, a large-capacity 
memory for storing image data is not required. In addition, a relative low- 
performance CPU, processing operation circuit or the like can be used. Therefore, 
the size and cost of an image processing system capable of recognizing an object 
can be significantly reduced. 

r00431 An object can be recognized from an image by a method for 
determining the shape of the object, a method for determining a difference in color of 
images, and a combination thereof. In the present invention, an object is recognized 
based on the color of its image. In this case, the image data conversion section 
converts the UYVY value of image data output by the image data output section to 
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the object information processing apparatus to an HSV value, and it is preferably 
determined whether or not image data is a part of an object to be extracted, based 
on the HSV value. 

[00441 Hue (H) and saturation (S) are typically represented by a color 
wheel having 0 degrees to 360 degrees. Hue is represented by an angle on the color 
wheel and saturation is represented by a size (distance) from the center of the color 
wheel. Therefore, a color can be specified as follows: for example, "a hue is between 
10 degrees to 20 degrees and a saturation is 10 or more". In contrast, a UV value 
represents color information in a two-dimensional coordinate system. It is possible to 
specify a color with a UV value. However, it is not easy to determine a UV value 
range which matches the sensitivity of a human. In order to specify a color range 
which matches the sensitivity of a human, the color range is preferably represented 
by an area on a disk defined by an angle and a size (radius). 

r00451 The UYVY value-HSV value conversion is performed using a 
conversion table. The UYVY value-HSV value conversion can be completed in a 
clock cycle of an image transfer clock signal and can be thus performed in real time 
without using a fast clock signal. 

[00461 Analog signals, such as images taken by a camera module or the 
like, contain noise or ambiguous signals. Therefore, it is preferable to provide a noise 
removal section which removes noise from the results of determination by the object 
determination section. 
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r00471 An object which the object determination section determines is to 
be extracted may be a part of an already detected object. Therefore, it is preferable 
that the object inclusion relation determination section determines whether the object 
is a part of the already detected object or a new object which has not been detected 
so far. The object inclusion relation determination section generates data indicating 
the coordinates having maximum X, the coordinates having minimum X, the 
coordinates having maximum Y, and the coordinates having minimum Y, of 
coordinates (X, Y). Objects can be recognized using data of the four coordinate 
points, thereby making it possible to further reduce the amount of data. 

[00481 The information of an object stored in the object information 
storing section can be read out by the processing operation section on a frame-by- 
frame basis. Therefore, the object information storing section (e.g., a resistor, etc.) 
stores only one frame of coordinate data of an object which is determined to be an 
object to be extracted, thereby making it possible to reduce the amount of data to be 
stored. 

[00491 The processing operation section reads out the coordinate data of 
an object from the object information storing section on a frame-by-frame basis, and 
the coordinate data of the object is compared between each frame, thereby making it 
possible to detect movement or change of the object. In this case, the coordinate 
data of an object can be temporarily retained in a resistor instead of a memory, so 
that the retained coordinate data can be compared with input data of the next frame 
to detect movement or change of an object. 
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rOOSOl Thus, the invention described herein makes possible the 
advantage of providing an object information processing apparatus capable of 
obtaining information required for object recognition in few types and In small 
quantities; an image processing system capable of relatively easily recognizing an 
object and detecting movement or change of an object using the object information; 
a game apparatus and an image processing method using the same. 

rOOSn These and other advantages of the present invention will become 
apparent to those skilled in the art upon reading and understanding the following 
detailed description with reference to the accompanying figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 

r00521 FIG. 1 is a block diagram showing a major configuration of an 
image processing system according to an embodiment of the present invention. 

[00531 FIG. 2 is a block diagram showing a major configuration of an 
object information extraction circuit of FIG. 1 . 

[00541 FIG. 3 is a flowchart showing a control flow of object information 
extraction by the object information extraction circuit of FIG. 2. 

[00551 FIG. 4 is a diagram for explaining an image data conversion of an 
image data conversion circuit of FIG. 2. 
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r00561 FIG. 5 is a block diagram showing a concrete exemplary 
configuration of an extraction condition determination circuit of FIG. 2. 

f 00571 FIG. 6 is a block diagram showing a major configuration of a noise 
removal circuit of FIG. 2. 

r00581 FIG. 7 is a circuit diagram showing a major configuration of the 
noise removal determination circuit of FIG. 2. 

r00591 FIG. 8 is a diagram for explaining an exemplary operation of a 
noise removal circuit of FIG. 2. 

[00601 FIG. 9 is a flowchart showing an exemplary processing flow in 
object inclusion relation determination in step S6 of FIG. 3. 

[00611 FIG. 1 0A is a diagram showing an exemplary manner of scanning 
image data according to an embodiment of this invention and a data state (A) of an 
object information storing register of the embodiment. 

[00621 Figure 10B is a diagram showing an exemplary manner of 
scanning image data according to an embodiment of this invention and a data state 
(B) of an object information storing register of the embodiment. 
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r00631 Figure 10C is a diagram showing an exemplary manner of 
scanning image data according to an embodiment of this invention and a data state 

(C) of an object information storing register of the embodiment. 

r00641 Figure 10D is a diagram showing an exemplary manner of 
scanning image data according to an embodiment of this invention and a data state 

(D) of an object information storing register of the embodiment. 

f00651 FIG. 10E is a diagram showing an exemplary manner of scanning 
image data according to an embodiment of this invention and a data state (E) of an 
object information storing register of the embodiment. 

r00661 FIG. 11 is a diagram showing an example in which movement or 
change of an object is detected by an image processing system according to an 
embodiment of the present invention. 

[00671 FIG. 12 is a diagram showing an example in which movement or 
change of an object is detected by an image processing system according to an 
embodiment of the present invention. 

r00681 FIG. 13 is a diagram showing an example in which movement or 
change of an object is detected by an image processing system according to an 
embodiment of the present invention. 
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f00691 FIG. 14 is a diagram showing an example in which movement or 
change of an object is detected by an image processing system according to an 
embodiment of the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

r00701 Hereinafter, the present invention will be described by way of 
illustrative examples with reference to the accompanying drawings. 

r0071 1 FIG. 1 is a block diagram showing a major configuration of an 
image processing system according to an embodiment of the present invention. 

r00721 In FIG. 1, an image processing system 1 comprises a camera 
module 2 as an image data output device, an object extraction LSI 3 as an object 
information processing device, and a one-chip microcomputer 4 which is a control^ 
device for controlling the camera module 2 and the object extraction LSI 3. 

r00731 The camera module 2 comprises a lens 21, an image pickup 
device 22 (e.g., a CCD, a CMOS imager, etc.), and an image processing DSP 23. 

r00741 Data needs to be communicated between the image processing 
DSP 23 and the one-chip microcomputer 4 in order to control camera module 2. The 
image processing DSP 23 performs communication via an interface circuit (not 
shown), such as aft a 12C bus or the like. Such an interface circuit is often not 
incorporated in the inexpensive one-chip microcomputer 4. In this embodiment, the 
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object extraction LSI 3 is provided with a camera module controlling bus interface 
circuit 31 described below. 

r00751 The object extraction LSI 3 comprises the camera module 
controlling bus interface circuit 31 for interfacing between the camera module 2 and 
the one-chip micro computer 4 and an object information extraction circuit 32. The 
object information extraction circuit 32 compares, for each pixel, a value indicated by 
color information contained in image data output by the camera module 2 with a 
reference value for an object to be extracted to determine whether or not an object is 
to be extracted. When it is determined that an object Is to be extracted, the 
coordinate data of the object to be extracted is stored. 

f00761 The object extraction LSI 3 has a small circuit scale, and 
therefore, can be incorporated into the image processing DSP 23 or the one-chip 
microcomputer 4 of the camera module 2. In this embodiment, the object extraction 
LSI 3 is a discrete unit so that a general-purpose camera module 2 and one-chip 
microcomputer 4 can be used to construct a system. 

r00771 The one-chip microcomputer 4 comprises a CPU 41 as a control 
section, a ROM 42 as a memory section, and the like. The ROM 42 stores an image 
processing control program. The camera module 2 and the object extraction LSI 3 
are controlled in accordance with the image processing control program stored in the 
ROM 42 to recognize an object contained in an image and detect movement or 
change of the object. 
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r00781 The one-chip microcomputer 4 is connected to an input setting 
section (not shown). The user appropriately enters Inputs via the input setting section 
to provide settings for the camera module 2 and the object extraction LSI 4. For the 
camera module 2, an image data output format, a gain, a white balance, and the like 
are set. For the object extraction LSI 3, reference values of the hue (H), saturation 
(8) and lightness (Y) of an object to be extracted, reference values for determining 
the inclusion relation of an object, and the like are set. For example, the following 
extraction condition (reference values) may be provided: "an object has a hue in the 
range from 20 degrees to 30 degrees, a saturation in the range of 50% or more, and 
a lightness in the range of 30% or more. Two objects are determined to be the same 
if the distance therebetw ee n there between is within 3 pixels". The number of 
extraction conditions is not limited to one. A plurality of extraction conditions may be 
provided. 

r00791 FIG. 2 is a block diagram showing a major configuration of the 
object information extraction circuit 32 of FIG. 1 . 

rOOSOl In FIG. 2, the object information extraction circuit 32 comprises an 
image data conversion circuit 321 as an image data conversion section, an object 
extraction condition determination circuit 322 as a determination section, a noise 
removal circuit 323 as a noise removal section, an object inclusion relation 
determination circuit 324 as an object inclusion relation determination section, and 
an object information storing register 325 as an object information storing section 
(object information storing section). 
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rOOSn The image data conversion circuit 321 format-converts the UYVY 
value of image data output by the camera module 2 to an HSV value. 

f 00821 In the extraction condition determination circuit 322, the HSV 
value of each pixel output by the image data conversion circuit 321 is compared with 
a reference value given to an object to be extracted to determine whether or not 
input image data contains an object to be extracted. 

[00831 The noise removal circuit 323 removes noise based on the result 
of determination by the extraction condition determination circuit 322. 

[00841 The object inclusion relation determination circuit 324 determines 
whether an object which has been determined to be an object to be extracted is a 
part of an object which has been already extracted or a new object which has not 
been extracted. The object inclusion relation determination circuit 324 produces 
coordinate data of the object. In this embodiment, the coordinate data of an object 
includes the coordinates having maximum X, the coordinates having minimum X, the 
coordinates having maximum Y and the coordinates having minimum Y, of the 
object. 

[00851 The object information storing register 325 stores the coordinate 
data of each object which has been determined to be an object to be extracted. In 
this embodiment, for example, one frame of coordinate data is stored in the object 
information storing register 325 for each object which has been determined to be an 
object to be extracted. 
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r00861 In this embodiment, a res i stor register is used as the object 
information storing section 325. The reason will be described below. 

f00871 Specifically, extracted object information is rewritten if required, 
and all extracted object information may be rewritten at a time. Therefore, a res i stor 
register is preferably used, in which all bits can be simultaneously assessed. Memory 
devices, such as RAM and the like, have high gate efficiency, however, the number 
of bits which can be simultaneously accessed via an I/O portion is limited. Therefore, 
in this embodiment, a r e s i stor register is used to construct an object information 
storing section. 

f00881 Hereinafter, an object information extraction method using the 
above-described object information extraction circuit 32 will be described. 

r00891 FIG. 3 is a flowchart showing a control flow of object information 
extraction by the object information extraction circuit 32 of FIG. 2. 

[00901 In step SI, image data (UYVY) output by the camera module 2 is 
input to the image data conversion circuit 321 of the object extraction LSI 3 on a 
pixel-by-pixel basis. 

rOOSI 1 In step 82, the image data format conversion is performed. In 
step S2, the image data conversion circuit 321 performs format conversion of the 
UYVY value of color information contained in input image data to an HSV value. The 
reason the UYVY value of input image data is converted to an HSV value is that a 
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hue (H) and a saturation (8) are represented by an angle and a size (distance) from 
the center of a color wheel ranging from 0 degrees to 360 degrees, and it is easy to 
specify a range based on the sensitivity of a human. 

r00921 Next, in step S3, extraction conditions are determined. In step S3, 
the extraction condition determination circuit 322 compares an HSV value output by 
the image data conversion circuit 321 with a reference value given to an object to be 
extracted on a pixel-by-pixel basis to determine whether or not input image data 
contains an object to be extracted. As can be seen from the flowchart, the object 
information extraction flow has no feedback due to a conditional branch or the like 
and proceeds to subsequent steps straightfonvardly. Therefore, it is possible to 
readily construct a circuit capable of performing extraction condition determination 
processes having different conditions in parallel. 

r00931 Inst e p In step S4, noise removal is performed. Instep S4, the 
noise removal circuit 323 removes noise from the result of determination by the 
extraction condition determination circuit 322. 

r00941 In step S5, it is determined whether or not data is a part of an 
object to be extracted, based on the result of determination by the extraction 
condition determination circuit 322 from which noise has been removed. If the data is 
determined to be a part of an object to be extracted in step S5 (Yes), the process 
goes to step S6. On the other hand, if the data is determined not to be a part of an 
object to be extracted in step S5 (No), the process goes to step S8 and the next pixel 
data is processed. 
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r00951 In step 86, object inclusion relation is determined. In step 86, The 
object inclusion relation determination circuit 324 determines whether an object 
which has been determined in step 85 to be an object to be extracted is a part of an 
object which has been already extracted or a new object which has not been 
extracted. The object inclusion relation determination circuit 324 produces coordinate 
data of the object. 

r00961 In step 87, values stored in the object information storing register 
325 are updated. The coordinate data of each object which has been determined to 
be an object to be extracted is newly retained. Thereafter, the process goes to step 
88 where the next pixel data is processed. 

r00971 Hereinafter, the processes in the components of FIG. 2 and the 
object extraction process flow of FIG. 3 will be described in greater detail. 

r00981 The image data format conversion in step 82 and the image data 
conversion circuit 321 of FIG. 2 will be described in detail. 

r00991 FIG. 4 is a diagram for explaining the image data conversion of 
the image data conversion circuit 321 of FIG. 2. 

rOOIOOl In FIG. 4, the image data conversion circuit 321 has a conversion 
table for converting a (U, V) value of input image data to an H value representing a 
hue and a conversion table for converting a (U, V) value of input image data to an 8 
value representing a saturation. An H value and an 8 value which have been 
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obtained by the image data conversion circuit 321 are supplied to the extraction 
condition determination circuit 322. A lightness (Y value) of image data input to the 
image data conversion circuit 321 is supplied as a V value to the extraction condition 
determination circuit 322. 

rOOIOn The above-described data conversion from a UYW value to an 
HSV value contains multiplication and the like. Therefore, the data conversion is not 
completed by one cycle if a conventional processing operation is employed. In 
contrast, in this embodiment, an image memory, such as a frame memory or the like, 
is not used, so that image data Is not stored in an image memory, and therefore, 
each process has to be performed in real time. To achieve this, an image data 
conversion method requiring a plurality of cycles is not preferable, and image data; 
conversion has to be completed in one cycle. In this embodiment, the image data 
conversion circuit 321 is provided with the conversion table between UYVY values 
and HSV values so that input values and output values have a one-to-one 
relationship, thereby making it possible to complete image data conversion in one- 
cycle. 

r001021 As used herein, the term "one cycle" refers to a cycle of a clock 
which is used for image data format conversion. In this embodiment, the object 
information extraction circuit 32 does not generate a clock and the image data format 
conversion is performed in synchronization with image data UYVY transferred from 
the image processing DSP 23 of the camera module 2. One cycle is herein defined 
based on a time point and period in which one pixel of image data is transferred 
unless othenA/ise dictated. 
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f001031 Next, the extraction condition determination in step S3 and the 
extraction condition determination circuit 322 of FIG. 2 will be described in detail. 

[001041 FIG. 5 is a block diagram showing a concrete exemplary 
configuration of the extraction condition determination circuit 322 of FIG. 2. 

r001051 In FIG. 5, the extraction condition determination circuit 322 
comprises a comparison circuit 322a and a multi-input AND circuit 322A for 
determining whether or not a condition 0 for object extraction is satisfied; a 
comparison circuit 322b and a multi-input AND circuit 322B for determining whether 
or not another condition 1 for object extraction is satisfied; a comparison circuit 322c 
and a multi-input AND circuit 322C for determining whether or not another condition 
2 for object extraction is satisfied; and so on. 

rOOIOSI Each of the comparison circuits 322a, 322b, 322c, . . . , 
determines whether or not each of a hue (H), a saturation (S) and a lightness (V) is 
within a predetermined value range. If the AND circuits 322A, 322B, 322C, . . . , 
determine that all the items satisfy the criteria, the AND circuit outputs a high-level 
output, i.e., it is determined that data satisfies a corresponding condition. The result 
of condition determination is represented by one-bit data which indicates whether or 
not the condition is satisfied. The one-bit data is transferred to the next process. The 
number of such bits is equal to the number of conditions. 



r001071 For example, in the case of "Condition 0, Lightness Vmin 
determination" in the comparison circuit 322a, a lightness (V) of input image data is 
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compared with "Vmin" which is a mininnum lightness value for the condition 0 which 
is provided as a reference value in a resistor of the object extraction LSI 3 in 
advance by the one-chip microcomputer 4. In this case, if the lightness value V of 
input image data is greater than the lightness Vmin for the condition 0, the condition 
0 is satisfied, so that the result of determination "1" is output to the AND circuit 322A. 
Similarly, condition determination is performed for hue H and saturation 8. 

[001081 As a result, for example, if the condition 0 is satisfied for all the 
items in the comparison circuit 322a, all the inputs to the AND circuit 322A are "1" 
and the output of the AND circuit 322A is also "1". The output "1" of the AND circuit 
322A indicates that currently input data satisfy the condition 0. Similarly, condition 1 , 
condition 2, are determined on a pixel-by-pixel basis, and the results of condition 
determination are supplied to the subsequent noise removal circuit 323. 

[001091 Next, noise removal in step S4 and the noise removal circuit 323 
of FIG. 2 will be described in detail. Note that analog signals, such as images taken 
by the camera module 2 or the like, contain various information and a large amount 
of noise or ambiguous signals. Digital processing encounters a problem of how to 
handle with noise or ambiguous signals. In this embodiment, for example, the 
extraction condition determination circuit 322 performs determination five times. Five 
determination results are retained in a shift r e s i stor register . If three or more of five 
pieces of data have the same determination results, it is determined that no noise is 
present and data is a part of an object to be extracted. 



rOOIIOl FIG. 6 is a block diagram showing a major configuration of the 
noise removal circuit 323 of FIG. 2. 
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r001in In FIG. 6, the noise removal circuit 323 comprises a shift rosictor 
register 15 for retaining determination results and a noise removal determination 
circuit 16 for determining whether or not data is noise, based on the retained five 
determination results. 

r001121 Data of the results of condition determination operations (the 
output of each AND circuit) is input from the extraction condition determination circuit 
322 to the shift r e s i stor register 15. An output terminal of each unit 15a to 15e (5 
units in FIG. 6) of the shift r e sistor register 15 is connected to the noise removal 
determination circuit 16. The result of noise removal by the noise removal 
determination circuit 16 is output as noise removed data to the object inclusion 
relation determination circuit 324. 

[001131 FIG. 7 is a circuit diagram showing a major configuration of the 
noise removal determination circuit 323 of FIG. 2. 

r001141 In FIG. 7, the noise removal determination circuit 16 comprises 
AND circuits 16a to 16j and an OR circuit 16k. The input terminals of the AND circuit 
16a are connected to the output terminals of from the third resistor unit 15c to the 
fifth resistor unit 15e. If all data in the resistor units 15c to 15e are "1", the AND 
circuit 16a outputs "1". Similarly, in the case of the AND circuits 16b to 16h, if all data 
in units (three units in FIG. 7) in the shift resistor 15 connected to input terminals of 
the AND circuit are "1", the AND circuit outputs "1". The output terminals of the AND 
circuits 16a to 16J are connected to the respective input terminals of the OR circuit 
16k. If at least one of the AND circuits 16a to 16j outputs "1", the OR circuit 16k 
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outputs "1". Therefore, if three or more of five pieces of data retained in the units of 
the shift resistor 15 are "1", the OR circuit 16k outputs "1" as data indicating the 
result of noise removal. In this case, it is determined that an object satisfying 
extraction conditions is present, but not noise. 

[001151 Hereinafter, an exemplary, actual, concrete operation of the noise 
removal circuit 323 of FIG. 7 will be described in detail. 

[001161 FIG. 8 is a diagram for explaining an exemplary operation of the 
noise removal circuit 323 of FIG. 2. 

[001171 A portion (a) of FIG. 8 indicates an initial state of the noise 
removal circuit 323. In the iftitaJ Initial state, all data in the five r e s i stor register units 
15a to 15e are "0", the noise removal determination circuit 16 outputs "0" as data 
indicating the result of noise removal. 

[001181 A portion (b) of FIG. 8 indicates that data is shifted by one unit, so 
that data "1" is read into the shift r e s i stor register 15 (state 1). In state 1, only the 
resistor unit 15a has data "1" among the five resistor units 15a to 15e, so that the 
processing result output by the noise removal determination circuit 16 Is still "0". 

[001191 A portion (c) of FIG. 8 indicates that another piece of data "1" Is 
read into the shift res i stor register 15 (state 2). A portion (d) of FIG. 8 indicates that 
another piece of data "0" is read into the shift r e sistor register 15 (state 3). In states 2 
and 3, two resistor units have data "1 "among the five resistor units 15a to 15e. In this 
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case, the noise removal determination circuit 16 outputs "0" as data indicating the 
processing result. 

[001201 A portion (e) of FIG. 8 indicates that another piece of data "1" is 
read into the shift r e s i stor register 15 (state 4), In state 4, three or more (here three) 
resistor register units have data "1" among the five r e sistor register units 15a to 15e. 
In this case, the noise removal determination circuit 16 outputs data "1" as data 
indicating the processing result. 

r0012n A portion (f) of FIG. 8 indicates that another piece of data "0" is 

read into the shift resistor register 15 (state 5). In state 5, three r e sistor register units 
have data "1" among the five fesistef register units 15a to 15e. In this case, the noise 
removal determination circuit 16 still outputs data "1" as data indicating the 
processing result. 

r001221 Note that the number of determination operations performed by 
the extraction condition determination circuit 322 is not limited to five. If the noise 
removal circuit 323 can determine whether or not data is a part of an object to be 
extracted, based on the number of r e s i stor register units having the same 
determination result, the number of r e s i stor register units in a shift resistor register 
and the number of determination operations by the extraction condition determination 
circuit 322 are not particularly limited. 



r001231 For example, if the number of determination operations is (2n-1) 
(odd number: n is an i nterg e r integer of 1 or more) and the majority of the (2n-1) 
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r e s i stor register units have the same determination result, it may be determined that 
data input to the noise removal circuit 323 is a part of an object to be extracted but 
not noise. If the number of determination operations is 7, the shift r e s i stor register 
contains at least 7 resistor units. 

r001241 For example, the number of determination operations is 2n (even 
number: n is an integer of 1 or more) and the majority of the 2n r e sistor register units 
have the same result, it may be determined that data input to the noise removal 
circuit 323 is a part of an object to be extracted but not noise. For example, the 
number of determination operations is 6, the shift r e sistor register contains at least 6 
r e sistor register units. 

r001251 Next, the object inclusion relation determination in step S6 and 
the object inclusion relation determination circuit 324 of FIG. 2 will be described in 
detail. 

[001261 If data appears which is determined to be a part of an object to be 
extracted by the processing and determination in steps S1 to S5, it is necessary to 
determine whether the data is a part of an object which has been already extracted 
or a new object which has not been extracted. In this embodiment, the object 
inclusion relation determination circuit 234 determines whether an object which has 
been determined to be an object to be extracted is a part of an object which has 
been already extracted or a new object which has not been extracted. 
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[001271 FIG. 9 is a flowchart showing an exemplary processing flow in 
object inclusion relation determination in step S6 of FIG. 3. 

r001281 As shown in FIG. 9, when an object has been extracted, the 
object information storing register 325 stores coordinate data indicating the location 
of the object, i.e., coordinate data indicating the coordinates on the image pickup 
device 22. In addition, if a plurality of conditions are provided, a condition matching 
flag indicating which condition is satisfied by an object is retained. 

1001291 For example, the following four pieces of coordinate data 

(coordinate information) are stored in the object information storing register 325: 

Coordinates having maximum X: (XXMAX, YXMAX), 
Coordinates having minimum X: (XXMIN, YXMIN), 
Coordinates having maximum Y: (XYMAX, YYMAX), and 
Coordinates having minimum Y: (XYMIN, YYMIN). 

[001301 In addition to the location information of an object, the size, shape 
or the like of an object may be roughly determined, thereby making it possible to 
detect the movement and change of the object. Only coordinate data of four points is 
stored for an object, whereby the amount of data can be significantly reduced as 
compared to when entire image data is stored. 

[001311 The image processing system 1 of this embodiment incorporates 
the object information storing register 325 for storing coordinate data, whose size is 
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predetermined by expecting the number of objects to be extracted, information of 
extracted objects is stored in the object information storing register 325 in order of 
chronology of extraction from the earliest. In FIG. 9, a pointer for indicating the 
ordinal number of a res i stor register for storing a newly extracted object, is 
represented by OBJ^POINT (OP). 

[001321 As described above, analog signals indicating images taken by 
the camera module 2 or the like contains a large amount of noise or ambiguous 
signals. Therefore, adjacent pixels do not necessarily provide continuous data in an 
object. When data indicating an object to be extracted (first data) is obtained, it is 
determined whether or not another piece of data satisfying the same conditions 
(second data) is present in the vicinity of the object. If the second data is present 
within a predetermined distance from the first data, the second data needs to be 
determined to be a part of the object. The predetermined distance is a distance 
criterion for determining that the first and second data are of the same object. The 
one-chip microcomputer 4 sets the distance criterion in a res i stor register of the 
object extraction LSI 3. The object inclusion relation determination circuit 324 
performs processing based on the distance criteria. In FIG. 9, the distance criterion is 
represented by (DX, DY). 

[001331 Assuming that an already detected object (first data) has the 
following coordinates: 

Coordinates having maximum X: (XXMAX, YXMAX), 
Coordinates having minimum X: (XXMIN, YXMIN), 
Coordinates having maximum Y: (XYMAX, YYMAX), and 
Coordinates having minimum Y: (XYMIN, YYMIN), 
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and that the location of another piece of data (second data) which is determined to 
satisfy the same conditions is represented by coordinates (X, Y), the coordinates of 
the second data are compared with the coordinates of the first data to determine 
whether or not the following expression is satisfied: 

XXMIN-DX<X<XXMAX+DX, and Y<YYMAX+DY. 

[001341 If the expression is satisfied, the first and second data are 
determined to be a part of the same object and the object information is updated. 
The update of the object information means that the minimum and/or maximum value 
of an X and/or Y coordinate are updated, and more specifically, data stored in the 
object information storing register 325 is updated to new minimum and/or maximum 
values. 

r001351 In FIG. 9, the comparison circuit 324a compares coordinate data 
indicating coordinates (X, Y) which have been determined to satisfy the condition 0 
(e.g., the output of the comparison circuit 324a is Yes: "1") with the coordinate data 
of an already detected object OBJECTED (e.g., an object satisfying the condition 0) 
using the comparison circuits 324b to 324d. If the coordinates (X, Y) satisfies the 
above-described expression, i.e., 

XXMIN^O-DX< X <XXMAX..l)+DX 



r001361 (e.g., both of the outputs of the comparison circuit 324b and the 
comparison circuit 324c are Yes: "1 "), and 
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Y<YYMAX.sub_0+DY 

[001371 (e.g., the output of the comparison circuit 324d is Yes: "1"), the 
AND circuit 324e determines that the data indicating the coordinates (X, Y) is a part 
of the object OBJECT_0 (e.g., the output of the AND circuit 324e is "1 "). 

r001381 Similarly, the determination as described above is performed for 
other objects 0BJECT_1, 0BJECT_2 (e.g., objects corresponding to the condition 1, 

the condition 2) The output of the AND circuit 324e is connected to the inputs of 

the AND circuit 324i to 324k. 

[001391 If the coordinate data of the coordinate: (X, Y) Is compared with 
the coordinate data YYMAX_0 of the already detected object OBJECT_0 using the 
comparison circuits 324f to 324h and 

Y>YYMAX._0 

r001401 (e.g., the output of the comparison circuit 324f is Yes: "1" and the 
maximum value of the Y coordinate is updated) and the output of the AND circuit 
324e is "1", the output of the AND circuit 324k is "1" and the coordinate data 
YYMAX_0 of the object OBJECT_0 in the object information storing register 325 is 
updated to Y. 



If X>XXMAX._0 
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r0014n (e.g., the output of the comparison circuit 324g is Yes: "1" and the 
maximum value of the X coordinate is updated) and the output of the AND circuit 
324e is "1", the output of the AND circuit 324j is "1", the coordinate data XXMAX_0 
of the object OBJECT_0 in the object information storing register 325 is updated to 
X. 

If X<XXMIN_0 

r001421 (e.g., the output of the comparison circuit 324h is Yes: "1" and the 
minimum value of the X coordinate is updated) and the output of the AND circuit 
324e is "1", the output of the AND circuit 3241 is "1" and the coordinate data 
XXMIN_0 of the object OBJECT_0 in the object information storing register 325 is 
updated to X. 

r001431 If the coordinate data indicating the coordinates (X, Y) matches no 
objects which have been detected, the coordinate data is determined to be a part of 
a new object and information of the data is stored in a new resistor in the object 
information storing register 325. 

[001441 In FIG. 9, the coordinate data indicating the coordinates (X, Y) is 
compared with the coordinate data of the already detected objects OBJECT_0, 

OBJECT_1, 0BJECT_2, If the coordinate data indicating the coordinates (X, 

Y) matches no objects which have been detected (e.g., all inputs of a NOR circuit 
3241 are "0"), it is determined that the coordinates (X, Y) are not included in any 
already detected objects (e.g., the output of the AND circuit 324i is "0" In this case, X 
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is written into coordinate data XXMIN_OP and XXIVIAX.OP of an object 
OBJECT_OP and Y is written into YYMIN_OP and YYMAX^OP thereof, in the object 
information storing register 325. In addition, the OBJ_POINT is incremented to 
0BJ_P0INT+1 which is a pointer indicating a resistor register in which a new object 
is to be stored. 

fOOl 451 In the object information extraction circuit 32 of the object 
information extraction LSI 3, the processing flow fronri step St to step S8 is 
repeatedly performed while one frame of image data is transferred from the camera 
module 2. The above-described serial processing is performed every time one pixel 
of image data is input. Information is updated in real time in the object information 
storing register 325. Therefore, at the time when one frame of image data has been 
completely transferred, information of the final result of image extraction is already 
stored in the object information storing register 325. 

[001461 The above-described processing flow of object information 
extraction will be described by way of more concrete examples below. 

[001471 In FIGS. 10A to 10E, a portion (a) of each of FIGS. 10A to 10E 
shows an exemplary manner of scanning image data. A portion (b) thereof indicates 
data states (A) to (E) of the object information storing register 325. 

[001481 In Figure IDA, transfer of image data begins from a point having 
coordinates (X, Y) = (0, 0). The image data is successively transferred along the X 
axis. Afterimage data on one line along the X axis is completely transferred, 
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scanning is performed on the next Y coordinate: (one line along the X axis). In an 
initial state, the object information storing register 324 is empty, i.e., the entire 
resistor register is filled with "0". 

r001491 In FIG. 10B, the first data satisfying an extraction condition is 
detected at coordinates (2, 1). Since no object has been detected so far, the data is 
determined to be a part of a new object. In the object information storing register 
324, the coordinates (2, 1 ) are written into: 

Coordinates having maximum X: (XXMAX, YXMAX), 
Coordinates having minimum X: (XXMIN, YXMIN), 
Coordinates having maximum Y: (XYMAX, YYMAX), and 
Coordinates having minimum Y: (XYMIN, YYMIN). 

rOOISOV In this case, if a plurality of conditions are provided, a condition 
matching flag is employed In addition to the above-described information so as to 
indicate that the object satisfies the conditions. 

fOOISn Subsequently, in Figure 10C, data satisfying the object extraction 
condition is detected at coordinates (0, 2). In this case, the X coordinate value "0" of 
the coordinates are compared with XXMIN and XXMAX in the object information 
storing register 325 and the Y coordinate value "2" is compared with YYMIN and 
YYMAX. If the coordinates (0, 2) is located within an allowable range with respect to 
XXMIN, XXMAX, YYMIN and YYMAX, the data (coordinates (0, 2)) are determined 
to be a part of the same object already detected and information is updated in the 
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object information storing register 325. It is now assumed that the coordinates (0, 2) 
satisfy the above-described condition. 

[001521 In this case, the following data is updated to coordinates (0, 2): 

Coordinates having minimum X: (XXMIN, YXMIN), and 
Coordinates having maximum Y: (XYMAX, YYMAX). 

r001531 Similarly, in FIG. 10D, data satisfying the extraction condition is 
detected at coordinates (4, 3), and the following data is updated to coordinates (4, 3): 

Coordinates having maximum X: (XXMAX, YXMAX), and 
Coordinates having maximum Y: (XYMAX, YYMAX). 

r001541 Similarly, in FIG. 10E, data satisfying the object extraction 
condition is detected at coordinates (3, 5) and the following data is updated to the 
coordinates (3, 5): 

Coordinates having maximum Y: (XYMAX, YYMAX). 

rOOl 551 In this manner, every time data satisfying the extraction condition 
is detected, data is updated in the object information storing register 324. As a result, 
the following information of the object is stored: 
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Condition matching flag: 1 
Coordinates having minimum X: (0, 2), 
Coordinates having maximum X: (4, 3), 
Coordinates having minimurri Y: (2, 1), and 
Coordinates having maximum Y: (3, 5). 

r001561 The coordinates satisfying the conditions are located at the 
coordinate locations indicated by the coordinate data. If other objects are detected, a 
number of sets of coordinate data, which is the same as the number of the other 
objects, are stored in the object information storing register 324. 

r001571 The object information storing register 324 initializes the 
coordinate data before the next frame of image data is output from the camera 
module 2 to the object extraction LSI 3. Object information extraction is repeated 
from the start. Therefore, the object information stored in the object information 
storing register 324 is read out by the CPU 41 of the one-chip microcomputer 4 
before the initialization of the object information storing register 324. 

r001581 The CPU 41 of the one-chip microcomputer 4 detects the 
presence/absence of an object contained in an image, movement or change thereof, 
or the like, using object information extracted by the object information extraction LSI 
3. Thus, the image processing system is controlled. Only one frame of data can be 
used to recognize the size and location of an object satisfying certain conditions 
(hue, saturation, and lightness). By comparing the coordinate data of an object 
between frames, movement or change of the object can be detected. 
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r001591 Hereinafter, a method for detecting movement or change of an 
object using the information (coordinate data) of an extracted object will be described 
in detail. 

r001601 As described above, in the image processing system 1 of this 
embodiment, the object information storing register 324 stores information about an 
object contained in an image, i.e., object extraction conditions, coordinates having 
maximum X, coordinates having minimum X, coordinates having maximum Y, and 
coordinates having minimum Y. 

[001611 This object information is read from the object information storing 
register 324 into a memory (e.g., RAM (not shown)) or a register of the one-chip 
microcomputer 4 for each frame. By comparing this object information with another 
frame of data, movement or change of an object can be detected. The information 
amount of an extracted object is significantly smaller than preprocessed image data. 
Therefore, the capacity of a memory or resistor required for storing data and the 
processing load of the CPU 41 can be significantly reduced. 

rOOl 621 For example, it is assumed that images having a screen size of 
352 pixelsx288 pixels are processed. If a screen (frame) of image data is 
conventionally stored, the following amount of data needs to be handled: 



YUV (8 bits for each) x352x288=51904512 bits. 
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r001631 In contrast, if only the results of object extraction are stored in this 
embodiment, the amount of data to be handled is: 

(the number of color conditions + (the number of bits for representing coordinate X + 
the number of bits for representing coordinate Y) x the number of coordinate points) 
X the number of objects. 

r001641 For example, it is assumed that up to 10 objects are detected 
under four color conditions and the screen size is 352 pixelsx288 pixels. In this case, 
one coordinate is represented by: 

9 bits (coordinate X) +9 bits (coordinate Y) =1 8 bits. 

[001651 In this embodiment, four coordinate points (coordinates having 
minimum X, coordinates having maximum X, coordinates having minimum Y, and 
coordinates having maximum Y) are stored for one object, the amount of the 
coordinate data is: 

(9 bits (coordinate X)+9 bits (coordinate Y))x4 coordinate points)=72 bits. 

r001661 Here, the number of color conditions corresponds to 4 bits. 
Therefore, one object requires: 

(4 bits (the number of color conditions) + (9 bits (coordinate X) +9 bits (coordinate Y)) 
x4 coordinate points) =76 bits. 
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r00167l Further, the number of objects to be recognized is assumed to be 
1 0. The total data size is: 

(4 bits (the number of color conditions) + (9 bits (coordinate X) +9 bits (coordinate Y)) 
x4 coordinate points) x 10=760 bits. 

r001681 Therefore, if it is assumed that up to 10 objects are detected 
under four color conditions at a time, the total data size is in this embodiment: 

(4+(9+9)x4)x1 0=760 bits. 

r001 691 Therefore, according to the image processing system 1 of this 
embodiment, the data amount per screen can be reduced as compared to the above- 
described conventional data amount by: 

760/51 90451 2.congruent.0.00001 46=0.001 46%. 

r001701 The difference is increased with an increase in screen size. 

r0017n For example, in the case of a screen size of 640 pixelsx480 
pixels, the data amount per screen is compared with the conventional data amount 
as follows: 

(4+(1 0+9)x4)x 1 0)/(8x8x8x640x480) 

=800/1 - 57286400.congruent.0.000005=0.0005%. 
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[001721 Thus, according to the image processing system 1 of this 
embodiment, the capacity of a memory or r e s i stor register for storing information 
about an extracted object can be significantly reduced. 

r001731 According to the image processing system 1 of the present 
invention, object extraction information for each frame, which has been read out from 
the object information storing register 324 of the object information extraction circuit 
32 and stored In a r e s i stor register or memory of the one-chip microcomputer 4, can 
be used to compare the location (coordinates) and size of an object under certain 
conditions (hue, saturation, lightness) between frames. Thereby, movement of an 
object can be recognized. 

r001741 FIGS. 11 to 14 are diagrams showing an example in which the 
image processing system 1 of this embodiment is used to detect movement or 
change of an object. 

r001751 In a detection example 1 of FIG. 11, the extraction information of 
an object A is compared between frame 1 to frame 3, the coordinate points of the 
object A is changed toward a direction indicated by an arrow in FIG. 11. Thus, it is 
determined that the object A moves in the direction indicated by the arrow. 

[001761 In a detection example 2 of FIG. 12, as shown in a portion (a) of 
FIG. 12, the extraction information of an object B is compared between a frame 1 
and a frame 2. In this case, the center of the object B is not changed and the size of 
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the object B is expanded. Therefore, as shown in a portion (b) of FIG. 12, it can be 
determined that the object B is looming toward a viewing site (camera module 1). 

[001771 In a detection example 3 of FIG. 13, a portion (a) of FIG. 13 
shows a cylindrical object like a bat having an upper portion and a lower portion 
which are painted with different colors. In this case, the orientation of the cylindrical 
object can be determined by viewing the two colors. It is assumed that the cylindrical 
object is a bat. The object information (coordinate data) of the bat is compared 
between a frame 1 to a frame 3. As shown in a portion (b) of FIG. 13, by detecting 
changes in the orientation and location of the bat, it can be determined whether or 
not the bat is swung, what trajectory is drawn by the bat, how fast the bat is moved, 
what timing the bat is swung, or the like. Alternatively, in the case of a table tennis 
racket, if the front and back sides of a plate-like object are painted with different 
colors, it can be determined which side of the racket is viewed, based on the color 
viewed. 

r001781 In a detection example 4 of FIG. 14, it is assumed that a ball is 
imaged. As shown in a portion (a) of FIG. 14, a ball-like object is painted with two 
colors. By comparing the extraction information of the ball-like object between a 
frame 1 to a frame 3, the movement, rotation or the like of the ball can be determined 
based on the orientation, distance and location of the ball, as shown in a portion (b) 
of FIG. 14. 

[001791 In the above-described example, an object is painted with two 
colors. In addition, three, four, or more colors may be used. By adjusting the 
proportions of painted color areas, it is possible to obtain optimum object information. 
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The scope of applications of the image processing system 1 can be expanded into, 
for example, game apparatus or the like. 

[001801 In another application, by painting a cube with 6 different colors, 
the sides of the cube can be recognized based on color conditions. Therefore, such 
a cube can be recognized as a die without recognizing numerical symbols according 
to the image processing system 1 of this embodiment. 

f001811 As described above, according to the image processing system 1 
of this embodiment, an object in an image can be recognized. Therefore, the present 
invention can be applied to various input information as compared to input devices, 
such as a key board, a joystick, and the like. By painting an object to be extracted, 
such as a bat, a racket, a ball, a glove, or the like, with a particular color, various 
object information can be extracted by the object information extraction circuit 32. 
Therefore, games having many changes can be achieved by adjusting an image 
processing program read into the one-chip microcomputer 4 and color information 
(e.g., information indicating hue, information indicating saturation, and information 
indicating lightness) of an object to be extracted, without changing the hardware 
configuration of the image processing system 1. 

[001821 Although certain preferred embodiments have been described 
herein, it is not intended that such embodiments be construed as limitations on the 
scope of the invention except as set forth in the appended claims. Various other 
modifications and equivalents will be apparent to and can be readily made by those 
skilled in the art, after reading the description herein, without departing from the 
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scope and spirit of this invention. All patents, published patent applications and 
publications cited herein are incorporated by reference as if set forth fully herein. 

rOOl 831 As described above, according to the present invention, image 
data from an image data output section (e.g., a camera module, etc.) is serially 
processed by an object information processing apparatus on a pixel-by-pixel basis. 
Therefore, a large-capacity memory for storing image data is not required, as is 
different from conventional techniques. Thus, complicated calculation and processing 
flows are not required for object extraction. As a result, image processing can be 
performed in real time without using a clock signal faster than an image transfer 
clock signal and the load of image processing on a CPU can be minimized. 

r001841 According to the present invention, an image memory or a high- 
performance CPU for recognizing an object in an image, which are conventionally 
required, are not necessary. A relatively low-performance, inexpensive CPU, 
processing operation circuit or the like can be used. Therefore, it is possible to 
significantly reduce the size and cost of an image processing system having a 
function of recognizing an object, a function of detecting movement or change of an 
object. Thereby, it is possible to achieve an image processing system capable of 
recognizing an object in an image and detecting movement or change of an object, 
which can be incorporated into toys, game apparatuses or the like for which low price 
is required. 

f 001 851 Various other modifications will be apparent to and can be readily 
made by those skilled in the art without departing from the scope and spirit of this 
invention. Accordingly, it is not intended that the scope of the claims appended 
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hereto be limited to the description as set forth herein, but rather that the claims be 
broadly construed. 



